Dynamically assigning multi-modality circumstantial data to assistant action requests for correlating with subsequent requests

ABSTRACT

Implementations set forth herein relate to an automated assistant that uses circumstantial condition data, generated based on circumstantial conditions of an input, to determine whether the input should affect an action been initialized by a particular user. The automated assistant can allow each user to manipulate their respective ongoing action without necessitating interruptions for soliciting explicit user authentication. For example, when an individual in a group of persons interacts with the automated assistant to initialize or affect a particular ongoing action, the automated assistant can generate data that correlates that individual to the particular ongoing action. The data can be generated using a variety of different input modalities, which can be dynamically selected based on changing circumstances of the individual. Therefore, different sets of input modalities can be processed each time a user provides an input for modifying an ongoing action and/or initializing another action.

BACKGROUND

Humans may engage in human-to-computer dialogs with interactive softwareapplications referred to herein as “automated assistants” (also referredto as “digital agents,” “chatbots,” “interactive personal assistants,”“intelligent personal assistants,” “conversational agents,” etc.). Forexample, humans (which when they interact with automated assistants maybe referred to as “users”) may provide commands and/or requests usingspoken natural language input (i.e., utterances) which may in some casesbe converted into text and then processed, and/or by providing textual(e.g., typed) natural language input.

In some instances, an automated assistant may operate in a variety ofdifferent contexts in which multiple different users are present andattempting to interact with the automated assistant. However, because ofa lack of versatility of some automated assistants, guest users seekingto interact with automated assistants (e.g., to retrieve public dataand/or invoke non-access restricted actions) may experience issuesinteracting with the automated assistants. For instance, a guest userwho provides a request for an alarm go off at a particular time may notbe designated as the only user with permission to modify the alarm. As aresult, other guest users may be able to modify the alarm before thealarm goes off. These circumstances can be exacerbated when multipleguest users are attempting to edit certain respective actions they haveinitialized, but the automated assistant is unclear as to which user isattempting to modify a particular ongoing action. As a result,computational resources can be wasted when actions are misidentified andmodified inadvertently. Furthermore, when a user provides subsequentcommands for directing the automated assistant to an affected action,additional computational resources can be wasted.

SUMMARY

Implementations set forth herein relate to an automated assistant thatuses instances of circumstantial conditions in order to initializeand/or modify certain actions requested by a user. The automatedassistant can use circumstantial conditions to identify a user that isproviding a particular input, and correlating the particular input to anaction that was previously initialized under similar circumstantialconditions. In circumstances in which the automated assistant is goingto receive inputs from different users (e.g., “guest” users) in order toinitialize different actions, the automated assistant can dynamicallyselect modalities for detecting circumstantial conditions associatedwith the different users, as a context of such users' changes. In thisway, computational resources can be preserved by providing a variety ofcombinations of modalities that can be used to correlate a user to anaction. Furthermore, by allowing the automated assistant to dynamicallyselect modalities for correlating a user to an ongoing action, errorsthat arise from incorrectly assigning user commands to actions requestedby other users can be mitigated. Mitigating such errors can furtherpreserve computational resources that might otherwise be consumed whencorrecting any actions that are inadvertently initialized by suchincorrect assignments.

Implementations set forth herein relate to an automated assistant thatorchestrates various inputs from different users according tocircumstances in which such inputs are provided. In someimplementations, various circumstantial conditions can be correlated toa particular user and/or a particular action requested by thatparticular user. In this way, any subsequent inputs can be correlated toprevious circumstantial conditions in order for the automated assistantto generate a suitable response without delaying the response by, forexample, requesting that the non-authenticated user authenticatethemselves. This practice of bypassing explicit authentications thatwould otherwise be requested in response to certain user inputs canmitigate latency that can be exhibited between a request from a user anda responsive action from the automated assistant (e.g., an audibleand/or graphical response and/or controlling of a smart device).Furthermore, this practice can be especially useful in situations inwhich multiple users are interacting with the same automated assistantbut not requesting performance of any actions that uses private data.

For example, while a first guest user is within a home of another user,the first guest user can invoke an automated assistant from a computingdevice within the home of the other user in order to create remindersfor later in the day. In order to create the reminders, the first guestuser can provide a spoken utterance such as, “Assistant, remind me tocall my Dad at 3:00 PM today.” In response, the automated assistant cancause circumstantial data to be generated to characterize anycircumstantial conditions in which the first guest user provided thespoken utterance, as well as reminder data characterizing the requestedreminder. Furthermore, the automated assistant can cause thecircumstantial data and the reminder data to be stored (e.g., at thecomputing device) in correlation with each other.

The circumstantial conditions characterized by the circumstantial datacan include a trajectory in which the first guest user was walking whenthe spoken utterance was provided by the guest user. The trajectory canbe determined using audio data generated based on audio input to amicrophone array that is connected to the computing device. Thecircumstantial conditions characterized by the circumstantial data canadditionally or alternatively include image data characterizing anappearance of the first guest user when the spoken utterance wasprovided by the guest user. The image data can include an image itself,and/or a signature or embedding generated based on processing the image(e.g., using a face recognition machine learning model). Thecircumstantial conditions characterized by the circumstantial data canadditionally or alternatively include voice data, which can include asignature or embedding generated based on processing the spokenutterance (e.g., using a voice recognition machine learning model). Anysignatures or embeddings can be generated based on receiving permissionfrom the first guest user, and/or can optionally be transiently stored(e.g., locally on the computing device) for a short duration (e.g.,until a corresponding action is completed and/or for a thresholdduration of time). In these and other manners, the circumstantial datacan be used for determining a signature and/or embedding for use whendetermining whether a subsequent input to the automated assistant isbeing provided by the first guest user.

For example, another second guest user can provide a subsequent input tothe automated assistant such as, “Assistant, push my reminder back anhour.” Because of the relevance of subsequent input to the spokenutterance provided by the first guest user, the computing device mightotherwise cause the reminder for the first guest user to be modified.However, the automated assistant can identify one or more circumstantialconditions corresponding to the subsequent input in order to ensure thatthe correct reminder is modified per the subsequent input. For instance,the automated assistant can determine, e.g. with permission from thesecond guest user, that an appearance of the second guest usercorrelates to circumstantial data stored in association with a previousinput provided by the second guest user. An appearance of the secondguest user can correlate to circumstantial data stored in associationwith the previous input when at least one feature of the appearance ofthe second guest user is shared with at least one feature of theprevious input (e.g. when the second guest user is currently standing inthe same location that they were standing when a previous image of theroom was captured).

Additionally, or alternatively, the automated assistant can determinethat a voice signature of the second guest user also correlates to thecircumstantial data that is stored in association with the previousinput provided by the second guest user. Based on the correlation(s),the automated assistant can access action data corresponding to thereminder action previously requested by the second guest in order thatthe automated assistant can affect changes to the appropriate actiondata. For instance, in response to the subsequent input from the secondguest user, and based on the correlation between the appearance of thesecond guest user and/or voice signature of the second guest user, theautomated assistant can modify the action data, such that thecorresponding reminder is pushed “back an hour” in accordance with theinput from the second guest user, thereby avoiding inadvertentlymodifying the reminder of the first guest user.

In some implementations, a first user and a second user (e.g., guestusers) can be interacting with an automated assistant in a kitchen areaof their home in order that they might each be individually guidedthrough steps of their respective recipes. As an example, the first usercan be making an appetizer using an appetizer recipe being detailed viathe automated assistant, and the second user can be making an entréeusing an entrée recipe that is simultaneously being accessed by theautomated assistant. While interacting with the automated assistant andcooking their respective food items, each user can be moving about thekitchen area and changing their orientation with respect to a computingdevice through which the automated assistant is being accessed.Therefore, in order to identify a user that is providing an input to theautomated assistant at any given time, the automated assistant canselectively process data generated according to one or more inputmodalities.

As an example, when the first user provides a spoken utterance such as,“Assistant, how much flour does my recipe call for?” the automatedassistant can, with permission from the first user and the second user,process one or more images captured by the computing device when thefirst user provided the spoken utterance. Furthermore, the automatedassistant can process audio data characterizing at least a portion ofthe spoken utterance. The one or more images and/or the audio data canbe processed to identify a correlation between features ofcircumstantial conditions in which the first user provided the spokenutterance and any other circumstantial conditions in which a previousaction was requested.

In order to identify the correlation between features of thecircumstantial conditions in which the first user provided the spokenutterance and the previous action was requested by the first user (e.g.,“Assistant, render my appetizer recipe at the computing device.”), datagenerated using one or more input modalities can be processed. The datacan characterize circumstantial conditions in which the first userprovided the latest spoken utterance to the automated assistant, andother circumstantial conditions in which the first user provided aprevious spoken utterance to the automated assistant. For example, imagedata, captured when the first user provided the previous spokenutterance and the latest spoken utterance, can characterize a silhouette(e.g., the second user) that is turned away from the computing device.Furthermore, audio data captured by a microphone array connected to thecomputing device can indicate that a direction from which the previousspoken utterance and the latest spoken utterance were provided does notcorrespond to the location of the silhouette. Therefore, because bothspoken utterances are determined to have not been provided by theperson/silhouette in the images, but, rather, someone standing outside afield of view of the camera, the automated assistant can determine thatthe spoken utterances are from the same person (e.g., the first user).

Furthermore, in order to generate an accurate response to the latestspoken utterance from the first user, the automated assistant candetermine whether there is a previously performed action that wasrequested by the first user, and then correlate the latest spokenutterance to that previous action. For instance, the previouslyperformed action could include causing the automated assistant to renderthe appetizer recipe at a display interface of the computing device.Therefore, based on a correlation between the latest spoken utterance(e.g., “Assistant, how much flour does my recipe call for?”) and thecontent of the previously performed action (e.g., natural languagecontent of the appetizer recipe), the automated assistant can respond tothe first user with data that is based on the appetizer recipe. In thisway, operations related to requesting the first user (i.e., anon-authenticated user) to perform an authentication step can bebypassed, thereby further preserving computational resources, as well asmitigating latency between the first user providing the spoken utteranceand the automated assistant providing a response (e.g., “The appetizerrecipe calls for 2 cups of flour.”).

The above description is provided as an overview of some implementationsof the present disclosure. Further description of those implementations,and other implementations, are described in more detail below.

Other implementations may include a non-transitory computer-readablestorage medium storing instructions executable by one or more processors(e.g., central processing unit(s) (CPU(s)), graphics processing unit(s)(GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a methodsuch as one or more of the methods described above and/or elsewhereherein. Yet other implementations may include a system of one or morecomputers that include one or more processors operable to execute storedinstructions to perform a method such as one or more of the methodsdescribed above and/or elsewhere herein.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A and FIG. 1B illustrate an automated assistant that correlatesrequested actions to certain circumstantial conditions in order tobypass explicit authentication for each request.

FIG. 2 illustrates a system for using circumstantial condition data todetermine whether an input should affect an action that has beeninitialized by a particular user.

FIG. 3 illustrates a method for authenticating a user to allow the userto modify and/or initialize one or more actions based on a correlationbetween separate instances of similar circumstantial conditions.

FIG. 4 is a block diagram of an example computer system.

DETAILED DESCRIPTION

FIG. 1A and FIG. 1B illustrate a view 100 and a view 130, respectively,of an automated assistant 104 that correlates requested actions tocertain circumstantial conditions in order to bypass explicitauthentication for each request. The automated assistant 104 can beavailable and/or accessible via a computing device 102, which includeone or more automated assistant interfaces, such as one or moremicrophones and/or one or more display devices. A first user 112 and asecond user 114 can interact with the automated assistant 104 throughspoken inputs, which can be processed by a speech recognition module 108of the computing device 102.

Because each user may wish to modify their respective requested actions,the automated assistant 104 can bypass soliciting each user for expressauthentication in response to each respective user input.

As an example, the first user 112 can provide a spoken utterance 120such as, “Assistant, turn on the television to Science Show at 8:30 PMEST,” in order to cause the automated assistant 104 to activate atelevision 116 at a particular time and render a particular show. Thefirst user 112 can provide the spoken utterance 120 while standing at anangle from the computing device 102 in a room 118. In response toreceiving the spoken utterance 120, the computing device 102 can employa speech processing engine 108 to process audio data that is generatedby the computing device 102 for characterizing the spoken utterance 120.The speech processing engine 108 can determine that the first user 112provided an invocation phrase (e.g., “Assistant . . . ”), and, inresponse, cause the automated assistant 104 to be initialized at thecomputing device 102.

The computing device 102 can cache data characterizing variouscircumstantial conditions that were present when the user 112 providedthe spoken utterance 120. For example, the computing device 102 caninclude one or more interfaces and/or input devices for receiving inputsfrom users. At least one interface can include an audio interface thatcomprises one or more microphones. In some implementations, thecomputing device 102 can include a microphone array, and an audioprocessing engine of the computing device 102 can process audio datagenerated using the microphone array to determine a direction from whicha particular user is providing a spoken utterance. For example, inresponse to receiving the spoken utterance 120, the computing device 102and/or automated assistant 104 can determine that the first userprovided the spoken utterance 120 from a particular direction and/orangle relative to a reference position and/or angle (e.g., −45 degreesfrom center, and/or Estimated_Coordinates_Meters (1.43, 3.12, 1.33)).

Additionally, or alternatively, in response to receiving the spokenutterance 120, a camera of the computing device 102 can be used tocapture one or more images, with prior permission from the user 112, ofthe user 112 providing the spoken utterance 120. Additionally, oralternatively, in response to receiving the spoken utterance 120, acamera of the computing device 102 can be used to capture one or moreimages, with prior permission from the user 112, of an area around thecomputing device 102 before, during, and/or after the user 102 providedthe spoken utterance 120. Images of the user 112 and/or surroundings ofthe computing device 102 can be processed at the computing device 102 toidentify one or more circumstantial conditions corresponding to thespoken utterance 120 provided by the user 112. For example, in responseto receiving the spoken utterance 120, the computing device 102 canprocess image data and/or audio data to generate circumstantial dataindicating that the user 112 was standing at −30 degrees from thecomputing device 102 when the user provided the spoken utterance 120.Additionally, or alternatively, the circumstantial data can indicatethat the second user 114 was lying down at 0 degrees and did not performany speaking actions, or at least did not provide the spoken utterance120.

In some implementations, other circumstantial data can be generated inresponse to the second user 114 providing another spoken utterance 122.The other spoken utterance 122 can be, for example, “Also, turn on somewhite noise at 9:30 PM.” In response to receiving the other spokenutterance 122, the automated assistant 104 can cause the computingdevice 102 to generate data for establishing a setting that will resultin white noise at 9:30 PM. Furthermore, in response to receiving theother spoken utterance 122, the automated assistant 104 can cause thecomputing device 102 to generate other circumstantial datacharacterizing circumstantial conditions in which the second user 114provided the other spoken utterance 122.

As an example, the other circumstantial data can characterize one ormore different circumstantial conditions in which the second user 114provided the other spoken utterance 122. The other circumstantial datacan characterize a location and/or posture of the second user 114, alocation of the first user 112 relative to the second user, one or moremotions of the second user 114 before, during, and/or after the seconduser 114 provided the spoken utterance, one or more motions of the firstuser 112 before, during, and/or after the second user 114 provided thespoken utterance, an arrangement of a room 118 in which the second user114 provided the other spoken utterance 122, one or more actions beperformed by the computing device 102 before, during, and/or after thesecond user 114 provided the other spoken utterance 122, and/or anyother features of a circumstance in which the second user 114 providedthe other spoken utterance 122.

When the action(s) requested by each user has been initialized via theautomated assistant 104, the circumstantial data corresponding to eachaction can be stored at the computing device 102 and subsequentlyaccessible to the automated assistant 104. In this way, one or morenon-authenticated users (e.g., users that have not undertaken anauthentication process directed by the automated assistant 104) canmodify a respective action and/or request another action withoutundertaking explicit authentication at least based on the automatedassistant 104 having access to the various circumstantial data. Rather,the circumstantial data corresponding to each user and/or each actioncan include an embedding that can be used to authenticate a particularuser that is attempting to modify a previously requested action and/orinitialize another action via the automated assistant 104. As anexample, and as depicted in view 130 of FIG. 1B, the second user 114 canprovide another spoken utterance 132, subsequent to the spoken utterance122 depicted in view 100 of FIG. 1A. The other spoken utterance 132 canbe, for example, “Never mind . . . please undo that.” In someimplementations, in response to receiving the other spoken utterance132, the automated assistant 104 can cause the circumstantial conditionsengine 106 to generate circumstantial data characterizing circumstancesin which the second user 114 provided the other spoken utterance 132.The circumstantial data can be processed using one or more previouslygenerated user embeddings and/or other circumstantial data in order toidentify a particular user to whom the circumstantial data mostcorresponds.

As an example, because the second user 114 was the last user to providea spoken utterance to the automated assistant 104, the computing device102 can have the microphone array of the computing device 102 operateaccording to beamforming settings. The beamforming settings can allowaudio input, sourced from an area that includes the second user 114, tobe more coherent to the computing device 102 than other sources of audioin the room 118. The circumstantial data generated in response to theother spoken utterance 132 can characterize this beamforming arrangementand be used to authenticate this subsequent spoken utterance 132 fromthe second user 114 and/or identify an action to modify and/orinitialize in response to the other spoken utterance 132. For example,the automated assistant 104 can process the circumstantial datacorresponding to the spoken utterance 122 and the circumstantial datacorresponding to the spoken utterance 132 in order to determine whetherthere is a correspondence between these instances of circumstantialdata. Additionally, or alternatively, the automated assistant 104 canprocess the circumstantial data corresponding to the spoken utterance122 and the circumstantial data corresponding to the spoken utterance132 in order to determine whether there is a stronger correlationbetween these instances of circumstantial data than any othercircumstantial data.

When a correlation between the instances of circumstantial data isidentified, the automated assistant 104 can bypass requesting that thesecond user 114 confirm, through some gesture or other input, that theyprovided the request for the initializing the action being affected. Inother words, instead of the automated assistant 104 providing an outputsuch as, “Ok, please confirm the action you would like to undo, cancel,or amend,” and/or “Ok, please provide the invocation phrase in order tomodify the requested action.” Rather, based on determining a strongenough correlation between the instances of circumstantial data, theautomated assistant 104 can bypass providing such output and “undo” thepreviously requested action of generating white noise at a particulartime. Additionally, or alternatively, based on determining that there isa correlation between the instances of circumstantial data, theautomated assistant 104 can identify a previously performed action toaffect in accordance with the other spoken utterance 132. In this way,the automated assistant 104 can bypass soliciting and/or processingadditional inputs from the second user 114 for authenticating therequest and/or identifying the action (e.g., setting a time for whitenoise to play) to be modified.

FIG. 2 illustrates a system 200 for using conditional data, generatedbased on circumstantial conditions of an input to an automated assistant208, to determine whether the input should affect an action that hasbeen initialized by a particular user. The automated assistant 208 canoperate as part of an assistant application that is provided at one ormore computing devices, such as a computing device 202 and/or a serverdevice. A user can interact with the automated assistant 208 viaassistant interface(s) 204, which can be a microphone, a camera, a touchscreen display, a user interface, and/or any other apparatus capable ofproviding an interface between a user and an application. For instance,a user can initialize the automated assistant 208 by providing a verbal,textual, or a graphical input to the assistant interface to cause theautomated assistant 208 to perform a function (e.g., provide data,control a peripheral device, access an agent, generate an input and/oran output, etc.). The computing device 202 can include a display device,which can be a display panel that includes a touch interface forreceiving touch inputs and/or gestures for allowing a user to controlapplications of the computing device 202 via the touch interface. Insome implementations, computing device 202 can lack a display device,thereby providing an audible user interface output, without providing agraphical user interface output. Furthermore, the computing device 202can provide a user interface, such as a microphone, for receiving spokennatural language inputs from a user. In some implementations, thecomputing device 202 can include a touch interface and can be void of acamera, but can optionally include one or more other sensors.

The computing device 202 and/or other computing devices can be incommunication with the server device over a network, such as theinternet. Additionally, the computing device 202 and the other computingdevices can be in communication with each other over a local areanetwork (LAN), such as a Wi-Fi network. The computing device 202 canoffload computational tasks to the server device in order to conservecomputational resources at the computing device 202. For instance, theserver device can host the automated assistant 208, and computing device202 can transmit inputs received at one or more assistant interfaces 204to the server device. However, in some implementations, the automatedassistant 208 can be hosted at the computing device 202 as a clientautomated assistant.

In various implementations, all or less than all aspects of theautomated assistant 208 can be implemented on the computing device 202.In some of those implementations, aspects of the automated assistant 208are implemented via the client automated assistant of the computingdevice 202 and interface with the server device that implements otheraspects of the automated assistant 208. The server device can optionallyserve a plurality of users and their associated assistant applicationsvia multiple threads. In implementations where all or less than allaspects of the automated assistant 208 are implemented via a clientautomated assistant at the computing device 202, the client automatedassistant can be an application that is separate from an operatingsystem of the computing device 202 (e.g., installed “on top” of theoperating system)—or can alternatively be implemented directly by theoperating system of the computing device 202 (e.g., considered anapplication of, but integral with, the operating system).

In some implementations, the automated assistant 208 and/or the clientautomated assistant can include an input processing engine 212, whichcan employ multiple different modules for processing inputs and/oroutputs for the computing device 202 and/or the server device. Forinstance, the input processing engine 212 can include a speechprocessing module 214 that can process audio data received at anassistant interface 204 to identify the text embodied in the audio data.The audio data can be transmitted from, for example, the computingdevice 202 to the server device in order to preserve computationalresources at the computing device 202.

The process for converting the audio data to text can include a speechrecognition algorithm, which can employ neural networks, and/orstatistical models for identifying groups of audio data corresponding towords or phrases. The text converted from the audio data can be parsedby a data parsing engine 216 and made available to the automatedassistant as textual data that can be used to generate and/or identifycommand phrases from the user. In some implementations, output dataprovided by the data parsing engine 216 can be provided to a parameterengine 218 to determine whether the user provided an input thatcorresponds to a particular action and/or routine capable of beingperformed by the automated assistant 208 and/or an application or agentthat is capable of being accessed by the automated assistant 208. Forexample, assistant data 222 can be stored at the server device and/orthe computing device 202, as client data 206, and can include data thatdefines one or more actions capable of being performed by the automatedassistant 208 and/or client automated assistant, as well as parametersnecessary to perform the actions. Additionally, or alternatively, thecomputing device 202 can include one or more different applications 224,which can be accessed by the automated assistant 208, with priorpermission from a user, for performing various other actions. Parametersand/or slot values for performing such actions can be provided by a uservia input to one or more assistant interfaces 204 and/or stored asapplication data 232, client data 206, and/or assistant data 222.Furthermore, each status of each action can be stored at the computingdevice 202 in association with a particular user (e.g., a guest user)and/or one or more circumstantial conditions of an input provided by theparticular user. In this way, explicit authentication by the particularuser can be bypassed when the particular user provides subsequent inputsfrom a perspective having similar circumstantial conditions relative tothe one or more circumstantial conditions previously acknowledged by theautomated assistant 208.

As an example, a first user can provide a spoken utterance to theautomated assistant 208 in order to initialize a first timer that iscontrolled by an application of the applications 224. In response toreceiving the spoken utterance, the automated assistant 208 can employthe input processing engine 212 and/or an action processing engine 228to determine an action and/or an application that the first user isintending to invoke. In some implementations, the automated assistant208 can use the action processing engine 228 to determine whether arequested action is a restricted action or is a non-restricted action.For example, a restricted action can correspond to action restrictionsthat do not allow a guest user to control the action, whereas anon-restricted action is one that a guest user is allowed to control.The automated assistant 208 can determine that the request from thefirst user corresponds to an unrestricted action, and therefore theautomated assistant 208 can bypass authenticating the first user. Thiscan preserve computational resources that might otherwise be consumedprocessing data in furtherance of authenticating an identity of thefirst user.

Because the automated assistant 208 has elected to bypass authenticatingthe first user based on the requested action being an unrestrictedaction, the automated assistant 208 can use a circumstantial conditionsengine 226 to generate circumstantial data associated with the spokenutterance from the first user. The circumstantial conditions engine 226can process data from one or more assistant interfaces 204 in order togenerate circumstantial data that characterizes one or more propertiesof a circumstance in which the first user provided the spoken utterance.For example, one or more assistant interfaces 204 can include one ormore cameras, and image data generated using the one or more cameras canbe processed by the circumstantial conditions engine 226 to generatecircumstantial data. The circumstantial data can characterize: one ormore images of one or more persons within the field of view of thecamera when the first user provided the spoken utterance, a respectivesize of one or more faces have one or more persons within the field ofview of the camera, a trajectory of one or more persons within view ofthe camera, mouth movement and/or other bodily movement performed by oneor more persons within the field of view of the camera, posture and/orchanges in posture of one or more persons, an arrangement of a room inwhich the first user provided the spoken utterance, a time of day inwhich the user provided the spoken utterance, and/or any other featuresthat can be determined via images captured by a computing device. Anycombination of any features characterized by circumstantial data can beused by a guest embedding engine 230 to generate an embedding tocorrelate to a particular guest user (i.e., a non-authenticated user),input, and/or action.

In furtherance of the aforementioned example, conditional datacharacterizing circumstantial conditions in which the first userprovided the spoken utterance can include facial features and/or postureof the first user when the first user provided the spoken utterance. Insome implementations, this conditional data generated by thecircumstantial conditions engine 226 can be provided to a guestembedding engine 230 in order to generate an embedding to correlate tothe first user and/or the first timer action.

Subsequent to the first user providing the spoken utterance forinitializing the first timer, a second user can provide another spokenutterance for initializing a second timer. The second user can providethe spoken utterance without the first user knowing, thereby preventingthe first user from being on notice that the second timer has beeninitialized. In response to the automated assistant 208 receiving theother spoken utterance, the automated assistant 208 can cause theapplication 224 to initialize the second timer, and also generate otherconditional data characterizing circumstantial conditions in which thesecond user provided the spoken utterance. For example, the second usercan be outside of a room in which the computing device 202 is located,therefore, a microphone of the assistant interfaces 204 can captureaudio data corresponding to the other open utterance. Furthermore, acamera of the assistant interfaces 204 can capture image data that doesnot include the source of the audio, indicating that the second user isoutside of the room. The circumstantial conditions engine 226 canprocess this audio data and image data to generate additional data thatcharacterizes the second user as being outside of the room and/or havinga particular voice signature. The conditional data can be used by theguests embedding engine 230 in order to generate another guest embeddingfor the second user and/or the second timer action.

Subsequent to the second user providing the other spoken utterance andinitializing the second timer, the first user can provide a subsequentspoken utterance to the automated assistant 208 in order to cancel thefirst timer that first set. However, because the automated assistant 208did not authenticate an identity of the first user that established thefirst timer, the automated assistant in 208 can use the embeddingsstored in the assistant data 222, in order to process conditional datacharacterizing circumstances in which the first user provided thesubsequent spoken utterance, in order to identify the correct timer tocancel. For example, the automated assistant of 208 can cause thecircumstantial conditions engine 226 to generate conditional datacharacterizing circumstances in which the subsequent spoken utterancewas provided. This conditional data, along with one or more guestembeddings, can be processed in order to identify previouslyacknowledged circumstantial conditions that are most similar to the mostrecent circumstantial conditions corresponding to the subsequent spokenutterance.

For example, the circumstantial conditions corresponding to thesubsequent spoken utterance can indicate that a person standing to oneside of a field of view of a camera of the computing device 202 was thesource of the subsequent spoken utterance. This conditional data can beprocessed in order to identify a previously requested action to cancel.In some implementations, the circumstantial conditions engine 226 candetermine that the most recent circumstances are most similar toprevious circumstances in which the first user requested the first timerto be initialized while standing within a field of view of the camera.For example, the most recent circumstantial condition data can beprocessed with each embedding to generate a score for each for degree ofsimilarity between the most recent circumstantial conditions and eachprevious circumstantial condition. An action corresponding to a previouscircumstantial condition that is scored highest relative to any othercircumstantial condition can be selected for modifying in response tothe subsequent spoken utterance. As a result, the action correlated tothe most similar circumstantial condition can be canceled in response tothe subsequent spoken utterance from the first user. In this way,computational resources (e.g., processing bandwidth and networkbandwidth) that might otherwise be consumed when incorrectly cancelingcertain actions can be preserved.

In some implementations, circumstantial data for a particular user(e.g., guest user) can be updated as a user continues to interact withthe automated assistant and/or computing device. For example, when theuser is providing a spoken utterance to an automated assistant but isnot within a field of view of a camera that is associated with theautomated assistant, the automated assistant can cause circumstantialdata and/or an embedding to be generated that is based on thecircumstances (e.g., voice characteristics, acoustic properties) inwhich the user provided the spoken utterance. Thereafter, the user cancontinue to affect any of their previously requested actions using theirvoice, as long as the circumstantial conditions are similar enough tothe previously generated circumstantial data and/or embedding. However,should the user walk into the field of view of the camera and provideanother spoken utterance, visible features of the user can be identifiedin image data and used to update the circumstantial data and/orembedding corresponding to the user. In other words, because thecircumstantial conditions of the subsequent spoken utterance wouldcorrelate to the previous circumstantial data and/or embedding, anyimage data collected during the subsequent spoken utterance can be usedto update the circumstantial data and/or embedding. Thereafter, shouldthe user provide a gesture (e.g., hold their hand up to stop theautomated assistant from continuing to do something) withoutsimultaneously providing a spoken utterance, the gesture can becorrelated to the updated circumstantial and/or embedding. As a result,the gesture can affect one or more ongoing actions that may becorrelated to the user, the circumstantial data, and/or the embedding.

In some implementations, circumstantial data that is stored incorrelation with a particular user and/or a particular requested actioncan be transient and/or otherwise stored temporarily. For instance, thecircumstantial data corresponding to a particular action can be storedduring a time that the action is ongoing and/or until the action iscomplete, and/or a period of time subsequent to the action completing orotherwise ceasing. As an example, circumstantial data corresponding to aguest user (i.e., a non-authenticated user) requesting a timer to go offat a particular time can be stored until the timer is dismissed, oruntil a period of time after the timer is dismissed transpires.Additionally, or alternatively, circumstantial data and/or embeddingsfor a particular guest user can be purged after some amount of “idle”time and/or inactivity time has transpired. In this way, the automatedassistant would not be continually building an embedding for each guestuser, but would limit such embeddings to transient situations in which aguest user is requesting services of the automated assistant.Furthermore, such storing and purging of circumstantial data and/orembeddings can be performed exclusively at a client device.Alternatively, such storing and purging of circumstantial data and/orembeddings can be orchestrated at a client device and server device.

In some implementations, a threshold degree of similarities between aninstance of circumstantial data to an existing embedding can bedifferent depending on whether an embedding corresponds to a guest useror an authenticated user. An authenticated user can be a user that hasundergone steps for authenticating their identity with an automatedassistant and/or computing device by, for example, audibly repeating aphrase for speech authentication, orienting themselves for undergoingfacial recognition, and/or performing one or more other gestures infurtherance of generating an authenticated user embedding. Therefore,when certain circumstantial data is processed using various guestembeddings, a distance metric for each guest embedding can be generated.A particular guest user identifier can then be selected based on adetermination that the distance metric for a particular guest embeddingsatisfies a guest embedding threshold. The guest embedding threshold cancorrespond a greater distance relative to another distance forsatisfying an authenticated user embedding threshold. For example, thecircumstantial data that is generated for a particular circumstance inwhich a user is providing a spoken utterance can be processed usingguest embeddings and authenticated user embeddings. Although thecircumstantial data may satisfy a guest embedding threshold for aparticular guest embedding, the circumstantial data may not satisfy anauthenticated user embedding threshold. As a result, the spokenutterance will be considered to be provided by a guest user, and willnot be considered to be provided by a previously authenticated user.

FIG. 3 illustrates a method 300 for authenticating a user to allow theuser to modify and/or initialize one or more actions based on acorrelation between separate instances of similar circumstantialconditions. The method 300 can be performed by one or more computingdevices, applications, and/or any other apparatus or module capable ofinteracting with a user or automated assistant. The method 300 caninclude an operation 302 of determining whether an input to an automatedassistant has been detected. The input can be provided by a user to oneor more interfaces of a computing device. For example, the input can bea spoken utterance that is received at one or more microphones that arein communication with the computing device. The spoken utterance can be,for example, “Assistant, set a reminder to call my work in an hour.”When no input is detected for the automated assistant, the operation 302can be repeated until an input is received for the automated assistant.

The method 300 can further include an operation 304 of generatingcircumstantial data characterizing one or more circumstantial conditionsassociated with the input. The one or more circumstantial conditions canbe, but are not limited to, voice characteristics exhibited by the user,features of an appearance of the user, audio and/or visual details of anarea in which the user provided the input, a time at which the userprovided the input, a location from which the user provided the input,and/or any other information to which circumstantial conditions can beattributed.

The method 300 can further include an operation 306 of determiningwhether the circumstantial data corresponds to other circumstantialdata. The other circumstantial data can be generated prior to the userproviding the input to the automated assistant. Furthermore, the othercircumstantial data can be generated in response to another input fromthe user and/or another user. The other circumstantial data can alsocharacterize other circumstantial conditions such as, but not limitedto, voice characteristics exhibited by the person who provided the otherinput, features of the appearance of the person, audio and/or visualdetails of an area in which the other person provided the input, a timeat which the person provided the input, a location from which the personprovided the input, and/or any other information to which the othercircumstantial conditions can be attributed.

When the circumstantial data is determined to not correspond to theother circumstantial data, the method 300 can proceed from the operation306 to the operation 308. The operation 308 can include generating otheraction data characterizing another action requested via the input. Inother words, circumstantial data that is determined to not correspond tothe other circumstantial data, the input can be determined to be from adifferent user, thereby causing the automated assistant to initializeperformance of another action based on the input. The method 300 canfurther include an operation 310 of storing the other action data incorrelation with the circumstantial data generated at the operation 304.In this way, the other action data can be referenced subsequently,should the same user in the same circumstantial conditions provide asubsequent input in order to modify the other action and/or initialize adifferent action.

The method 300 can further include an operation 312 of performing theother action based on the input provided by the user. For example, inresponse to the user providing the input, “Assistant, set a reminder tocall my work in an hour,” the automated assistant can generate actiondata characterizing the reminder and store the action data incorrelation with the circumstantial data. In this way, should the userprovide a subsequent input modifying the reminder, the automatedassistant can compare any circumstantial data related to the subsequentinput to the circumstantial data stored in correlation with the actiondata. Additionally, this allows the user to bypass any steps forexplicit authorization before modifying the reminder, therebystreamlining action modifications and mitigating waste of computationalresources.

In some implementations, at the operation 306, the circumstantial datacan be determined to correspond to other circumstantial data that wasgenerated previous to the input being detected at the operation 302. Inother words, the same user that provided the input detected at theoperation 302, or a different user that provided the input detected atthe operation 302, may have provided a previous input that caused theother circumstantial data to be generated. Furthermore, this previousinput may have resulted in previous action data being generated andstored in association with the other circumstantial data. In furtheranceof acknowledging this previous action data, the method 300 can includean operation 314 of identifying the previous action data correspondingto the other circumstantial data.

In some instances, the previous action data can identify a previousaction requested by the user such as, for example, to establish thereminder to place the phone call. However, in other instances, the otheruser may have requested that a previously set reminder (e.g., a reminderto order food delivery) be modified. For example, the previous actionrequested by this other user may have been established by a spokenuttering such as, “Assistant, set a reminder to order delivery foodtonight.” Subsequently, this other user may have provided a separateinput such as, “Assistant, delete the reminder I set earlier.” However,because there have been two reminders established by two differentusers, the automated assistant will identify the reminder to modifybased on a correlation between instances of circumstantial data. Whenthe automated assistant has identified this correlation, the automatedassistant can identify the action requested by the other user, andtherefore perform any request from the other user in view of theprevious action.

The method 300 can proceed from the operation 314 to the operation 316,which can include performing an action based on the input and theprevious action data. The input can be the spoken utterance from theother user requesting that the automated assistant delete the reminderthat the other user set previously. Furthermore, the previous actiondata can identify the reminder that the other user set previously.Therefore, when performing the action based on the input, the automatedassistant can identify the reminder for ordering delivery as the targetof the subsequently requested action of deleting a reminder. However, itshould be noted that if the circumstantial data did not correspond toany other circumstantial data, the subsequent input requesting deletionof the reminder would have been characterized as not relating to anypreviously set reminder, and therefore would not have affected anypreviously set reminder. In this way, errors that might otherwise occurin multi-user (e.g., multiple guests) scenarios would be mitigated byimproving accuracy of correlations between various user requests andongoing actions.

In some implementations, the method 300 can include an optionaloperation 318 of determining whether there is a difference between thecircumstantial data and the other circumstantial data. When there is nodifference between the circumstantial data and the other circumstantialdata, and/or no difference that is exhibited to a degree that satisfiesa correlation threshold, the method 300 can proceed from the optionaloperation 318 to the operation 302. However, when a difference betweenthe circumstantial data and the other circumstantial data is identified,the method 300 can proceed from the optional operation 318 to theoptional operation 320.

The optional operation 320 can include generating supplementalcircumstantial data based on one or more identified differences betweencircumstantial data. In some implementations, the supplementalcircumstantial data can be the other circumstantial data that ismodified according to the one or more identified differences. In thisway, as one or more circumstantial conditions of a user change overtime, the circumstantial data associated with the user can be updatedaccordingly. This allows the automated assistant to authorize the userto provide inputs and modify ongoing actions without participating in aninterruptive authentication procedure that is secondary from the userproviding action request. This can preserve computational resources andnetwork resources, as fewer authentication inputs would be processed ata respective computing device.

FIG. 4 is a block diagram 400 of an example computer system 410.Computer system 410 typically includes at least one processor 414 whichcommunicates with a number of peripheral devices via bus subsystem 412.These peripheral devices may include a storage subsystem 424, including,for example, a memory 425 and a file storage subsystem 426, userinterface output devices 420, user interface input devices 422, and anetwork interface subsystem 416. The input and output devices allow userinteraction with computer system 410. Network interface subsystem 416provides an interface to outside networks and is coupled tocorresponding interface devices in other computer systems.

User interface input devices 422 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computer system 410 or onto a communication network.

User interface output devices 420 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 410 to the user or to another machine or computersystem.

Storage subsystem 424 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 424 may include the logic toperform selected aspects of method 300, and/or to implement one or moreof system 200, computing device 102, and/or any other application,device, apparatus, and/or engine discussed herein.

These software modules are generally executed by processor 414 alone orin combination with other processors. Memory 425 used in the storagesubsystem 424 can include a number of memories including a main randomaccess memory (RAM) 430 for storage of instructions and data duringprogram execution and a read-only memory (ROM) 432 in which fixedinstructions are stored. A file storage subsystem 426 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 426 in the storage subsystem 424, or inother machines accessible by the processor(s) 414.

Bus subsystem 412 provides a mechanism for letting the variouscomponents and subsystems of computer system 410 communicate with eachother as intended. Although bus subsystem 412 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computer system 410 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computer system 410depicted in FIG. 4 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputer system 410 are possible having more or fewer components thanthe computer system depicted in FIG. 4 .

In situations in which the systems described herein collect personalinformation about users (or as often referred to herein,“participants”), or may make use of personal information, the users maybe provided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current geographic location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. Also, certain data may be treated in one or more waysbefore it is stored or used, so that personal identifiable informationis removed. For example, a user's identity may be treated so that nopersonal identifiable information can be determined for the user, or auser's geographic location may be generalized where geographic locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular geographic location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and/or used.

While several implementations have been described and illustratedherein, a variety of other means and/or structures for performing thefunction and/or obtaining the results and/or one or more of theadvantages described herein may be utilized, and each of such variationsand/or modifications is deemed to be within the scope of theimplementations described herein. More generally, all parameters,dimensions, materials, and configurations described herein are meant tobe exemplary and that the actual parameters, dimensions, materials,and/or configurations will depend upon the specific application orapplications for which the teachings is/are used. Those skilled in theart will recognize, or be able to ascertain using no more than routineexperimentation, many equivalents to the specific implementationsdescribed herein. It is, therefore, to be understood that the foregoingimplementations are presented by way of example only and that, withinthe scope of the appended claims and equivalents thereto,implementations may be practiced otherwise than as specificallydescribed and claimed. Implementations of the present disclosure aredirected to each individual feature, system, article, material, kit,and/or method described herein. In addition, any combination of two ormore such features, systems, articles, materials, kits, and/or methods,if such features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

In some implementations, a method implemented by one or more processorsis set forth as including operations such as determining, at a computingdevice, that a user has provided an input to an interface of thecomputing device, wherein the computing device provides access to anautomated assistant via the interface and the input is provided by theuser in furtherance of causing the automated assistant to perform anaction. The operations can further include generating, based on the userproviding the input to the computing device, circumstantial datacharacterizing one or more circumstantial conditions associated with theuser requesting performance of the action. The operations can furtherinclude determining, based on the circumstantial data, whether the oneor more circumstantial conditions correlate to one or more othercircumstantial conditions, wherein the one or more other circumstantialconditions are characterized by other circumstantial data that is storedat the computing device in correlation with a previous action that wasperformed by the automated assistant before the user provided the inputand in response to a previous request from a non-authenticated user. Theoperations can further include, when the one or more circumstantialconditions are determined to correlate to the one or more othercircumstantial conditions: accessing, based on determining that the oneor more circumstantial conditions correlate to the one or more othercircumstantial conditions, action data that is associated with theprevious action that was previously performed in response to theprevious request from the non-authenticated user, and causing, inresponse to the input to the interface of the computing device, theautomated assistant to perform the action using the action data.

In some implementations, the operations can further include, when theone or more circumstantial conditions are determined to not correlate tothe one or more other circumstantial conditions: generating, based ondetermining that the one or more circumstantial conditions do notcorrelate to the one or more other circumstantial conditions, otheraction data characterizing the action requested by the user via theinput. In some implementations, the operations can further include, whenthe one or more circumstantial conditions are determined to notcorrelate to the one or more other circumstantial conditions: storing,based on determining that the one or more circumstantial conditions donot correlate to the one or more other circumstantial conditions, otheraction data in correlation with the circumstantial data at the computingdevice.

In some implementations, the one or more circumstantial conditions andthe one or more other circumstantial conditions are characterized usingseparate inputs to separate input modalities of the computing device. Insome implementations, the circumstantial data is generated using acombination of one or more input modalities of the computing device andthe other circumstantial data is generated using a different combinationof one or more input modalities of the computing device. In someimplementations, the interface is a microphone array that is incommunication with the computing device, the one or more othercircumstantial conditions are associated with a location of the user,and the method further comprises: when the one or more circumstantialconditions are determined to correlate to the one or more othercircumstantial conditions: causing, based on determining that the one ormore circumstantial conditions correlate to the one or more othercircumstantial conditions, the microphone array to operate in accordancewith a beam forming technique in furtherance of isolating subsequentspoken utterances from the user.

In some implementations, the interface is a microphone array that is incommunication with the computing device, and the one or morecircumstantial conditions include an image characteristic that isidentified based on a light input to a camera of the computing device,and the one or more other circumstantial conditions include a positioncharacteristic that is identified based on audio input to the microphonearray. In some implementations, the operations can further include, whenthe one or more circumstantial conditions are determined to correlate tothe one or more other circumstantial conditions: determining that theuser is the non-authenticated user who provided the previous request,determining, based on determining that the one or more circumstantialconditions correlate to the one or more other circumstantial conditions,a difference between the one or more circumstantial conditions and theone or more other circumstantial conditions, and generating supplementalcircumstantial data based on the difference between the one or morecircumstantial conditions and the one or more other circumstantialconditions.

In other implementations, a method implemented by one or more processorsis set forth as including operations such as determining, at a computingdevice, that a user has provided a first input to an interface of thecomputing device, wherein the computing device provides access to anautomated assistant via the interface and the first input is provided bythe user in furtherance of causing the automated assistant to perform afirst action. The operations can further include generating, based onthe user providing the first input to the computing device, firstcircumstantial data characterizing a first circumstantial condition inwhich the user requested performance of the first action. The operationscan further include determining, at the computing device, that anotheruser has provided a second input to the interface of the computingdevice, wherein the second input is provided by the other user infurtherance of causing the automated assistant to perform a secondaction. The operations can further include generating, based on theother user providing the second input to the computing device, secondcircumstantial data characterizing a second circumstantial condition inwhich the other user requested performance of the second action. Theoperations can further include, subsequent to the user providing thefirst input and the other user providing the second input: determiningthat a third input has been provided to the computing device infurtherance of causing the automated assistant to perform a thirdaction, generating, based on the third input being provided to thecomputing device, third circumstantial data characterizing a thirdcircumstantial condition in which the third input was provided to thecomputing device, and determining, based on the third circumstantialdata, whether the third circumstantial condition in which the thirdinput was provided to the automated assistant correlates to the firstcircumstantial condition or the second circumstantial condition. Theoperations can further include, when the third circumstantial conditionis determined to correlate to the first circumstantial condition:accessing, in response to determining that the third circumstantialcondition correlates to the first circumstantial condition, first actiondata that is associated with the first action, and causing, in responseto the third input to the automated assistant, the automated assistantto provide a responsive output that is based on the first action data.

In some implementations, the first circumstantial condition and thesecond circumstantial condition are characterized using separate inputsto separate input modalities of the computing device. In someimplementations, the first circumstantial data is generated using one ormore input modalities of the computing device and the secondcircumstantial data is generated using other input modalities of thecomputing device. In some implementations, the interface is a microphonearray that is in communication with the computing device, and the firstcircumstantial condition includes an image characteristic that isidentified based on a light input to a camera of the computing device,and the second circumstantial condition includes a positioncharacteristic that is identified based on audio input to the microphonearray. In some implementations, the interface is a microphone array thatis in communication with the computing device, the first circumstantialcondition is associated with a location of the user, and the methodfurther comprises: when the third circumstantial condition is determinedto correlate to the first circumstantial condition: causing, based ondetermining that the one or more circumstantial conditions correlate tothe one or more other circumstantial conditions, the microphone array tooperate in accordance with a beam forming technique in furtherance ofisolating subsequent spoken utterances from the user.

In yet other implementations, a method implemented by one or moreprocessors is set forth as including operations such as determining, ata computing device, that a user has provided an input to an interface ofthe computing device, wherein the computing device provides access to anautomated assistant via the interface. The operations can furtherinclude generating, based on the user providing the input to thecomputing device, circumstantial data characterizing one or morecircumstantial conditions associated with the input. The operations canfurther include determining, based on the circumstantial data, whetherthe one or more circumstantial conditions correlate to one or more othercircumstantial conditions, wherein the one or more other circumstantialconditions are characterized by other circumstantial data that is storedat the computing device in correlation with a previous input that wasprovided in furtherance of the automated assistant performing a previousaction before the user provided the input. The operations can furtherinclude, when the one or more circumstantial conditions are determinedto correlate to the one or more other circumstantial conditions:accessing, based on determining that the one or more circumstantialconditions correlate to the one or more other circumstantial conditions,action data that is associated with the previous action, and causing, inresponse to the input to the interface of the computing device, theautomated assistant to perform the action using the action data.

In some implementations, the operations can further include, when theone or more circumstantial conditions are determined to not correlate tothe one or more other circumstantial conditions: generating, based ondetermining that the one or more circumstantial conditions do notcorrelate to the one or more other circumstantial conditions, otheraction data characterizing an action requested by the user via theinput. In some implementations, the operations can further include, whenthe one or more circumstantial conditions are determined to notcorrelate to the one or more other circumstantial conditions: storing,based on determining that the one or more circumstantial conditions donot correlate to the one or more other circumstantial conditions, otheraction data in correlation with the circumstantial data at the computingdevice.

In some implementations, the one or more circumstantial conditions andthe one or more other circumstantial conditions are characterized usingseparate inputs to separate input modalities of the computing device. Insome implementations, the circumstantial data is generated using acombination of one or more input modalities of the computing device andthe other circumstantial data is generated using a different combinationof one or more input modalities of the computing device. In someimplementations, the interface is a microphone array that is incommunication with the computing device, the one or more othercircumstantial conditions are associated with a location of the user. Insome implementations, the operations further include, when the one ormore circumstantial conditions are determined to correlate to the one ormore other circumstantial conditions: causing, based on determining thatthe one or more circumstantial conditions correlate to the one or moreother circumstantial conditions, the microphone array to operate inaccordance with a beam forming technique in furtherance of isolatingsubsequent spoken utterances from the user. In some implementations, theinterface is a microphone array that is in communication with thecomputing device, and the one or more circumstantial conditions includean image characteristic that is identified based on a light input to acamera of the computing device, and the one or more other circumstantialconditions include a position characteristic that is identified based onaudio input to the microphone array.

We claim:
 1. A method implemented by one or more processors, the methodcomprising: receiving audio data that is captured via one or moremicrophones of a computing device, wherein the audio data captures aspoken utterance, of a user, that is directed to an automated assistantof the computing device; determining, based on processing the audiodata: that the spoken utterance requests performance, by the automatedassistant, of an action that is of an ongoing action type, and that theuser is non-authenticated; causing performance of the action in responseto determining that the spoken utterance requests performance of theaction; in response to determining that the user is non-authenticatedand in response to determining that the action is of the ongoing actiontype: storing, only transiently until one or more purge conditions aresatisfied, circumstantial data and an association of the circumstantialdata to the action, the circumstantial data characterizing one or morecircumstantial conditions associated with the user providing the spokenutterance; receiving, at the computing device and prior to the one ormore purge conditions being satisfied, further user interface input thatrequests modification of the action; in response to the further userinterface input requesting modification of the action: comparing thecircumstantial data, stored in association with the action, to furthercircumstantial data characterizing one or more circumstantial conditionsassociated with the further user interface input; and determining, independence on the comparing, whether to modify the action in accordancewith the further user interface input.
 2. The method of claim 1, furthercomprising: monitoring for occurrence of the one or more purgeconditions; and in response to determining the occurrence of the one ormore purge conditions: purging the circumstantial data and theassociation of the circumstantial data to the action.
 3. The method ofclaim 2, wherein the one or more purge conditions comprise completion ofthe action.
 4. The method of claim 2, wherein the one or more purgeconditions comprise transpiring of a period of time after completion ofthe action.
 5. The method of claim 2, wherein the storing is at onlylocal storage of the computing device.
 6. The method of claim 1, furthercomprising: generating the circumstantial data based on the audio data.7. The method of claim 6, further comprising: generating thecircumstantial data further based on image data captured using one ormore cameras of the computing device.
 8. The method of claim 7, furthercomprising: identifying the image data, to utilize in generating thecircumstantial data, based on it being captured during the spokenutterance.
 9. The method of claim 1, wherein comparing thecircumstantial data to the further circumstantial data comprisesdetermining a difference measure between the circumstantial data and thefurther circumstantial data; and wherein determining, in dependence onthe comparing, whether to modify the action in accordance with thefurther user interface input comprises determining whether to modify theaction in dependence on whether the difference satisfies a threshold.10. The method of claim 9, further comprising, in response to thedifference satisfying the threshold: modifying the action in accordancewith the further user interface input.
 11. A computing device,comprising: one or more microphones; memory storing instructions; one ormore processors that execute the instructions to: receive audio datathat is captured via the one or more microphones and that captures aspoken utterance, of a user, that is directed to an automated assistantof the computing device; determine, based on processing the audio data:that the spoken utterance requests performance, by the automatedassistant, of an action that is of an ongoing action type, and that theuser is non-authenticated; cause performance of the action in responseto determining that the spoken utterance requests performance of theaction; in response to determining that the user is non-authenticatedand in response to determining that the action is of the ongoing actiontype: store, locally in the memory or other storage of the computingdevice and only transiently until one or more purge conditions aresatisfied, circumstantial data and an association of the circumstantialdata to the action, the circumstantial data characterizing one or morecircumstantial conditions associated with the user providing the spokenutterance; receive, at the computing device and prior to the one or morepurge conditions being satisfied, further user interface input thatrequests modification of the action; in response to the further userinterface input requesting modification of the action: compare thecircumstantial data, stored in association with the action, to furthercircumstantial data characterizing one or more circumstantial conditionsassociated with the further user interface input; and determine, independence on the comparing, whether to modify the action in accordancewith the further user interface input.
 12. The computing device of claim11, wherein, in executing the instructions, one or more of theprocessors are further to: monitor for occurrence of the one or morepurge conditions; and in response to determining the occurrence of theone or more purge conditions: purge the circumstantial data and theassociation of the circumstantial data to the action.
 13. The computingdevice of claim 12, wherein the one or more purge conditions comprisecompletion of the action.
 14. The computing device of claim 12, whereinthe one or more purge conditions comprise transpiring of a period oftime after completion of the action.
 15. The computing device of claim11, wherein, in executing the instructions, one or more of theprocessors are further to: generate the circumstantial data based on theaudio data.
 16. The computing device of claim 15, wherein, in executingthe instructions, one or more of the processors are further to: generatethe circumstantial data further based on image data captured using oneor more cameras of the computing device.
 17. The computing device ofclaim 16, wherein, in executing the instructions, one or more of theprocessors are further to: identify the image data, to utilize ingenerating the circumstantial data, based on it being captured duringthe spoken utterance.
 18. The computing device of claim 11, wherein, inexecuting the instructions, one or more of the processors are furtherto: generate the circumstantial data based on image data captured usingone or more cameras of the computing device.
 19. The computing device ofclaim 11, wherein in comparing the circumstantial data to the furthercircumstantial data one or more of the processors are to determine adifference measure between the circumstantial data and the furthercircumstantial data; and wherein in determining, in dependence on thecomparing, whether to modify the action in accordance with the furtheruser interface input one or more of the processors are to determinewhether to modify the action in dependence on whether the differencesatisfies a threshold.
 20. The computing device of claim 19, wherein, inexecuting the instructions, one or more of the processors are furtherto, in response to the difference satisfying the threshold: modify theaction in accordance with the further user interface input.